PARAMETER EFFICIENCY

Diversifying the Expert Knowledge for Task-Agnostic Pruning in Sparse Mixture-of-Experts (2025.findings-acl)

Challenge:	Pre-trained language models are language models that are pre-taught on large-scaled corpora in a self-supervised fashion.
Approach:	This tutorial provides a broad and comprehensive introduction to pre-trained language models . it focuses on emerging methods that enable PLMs to perform diverse downstream tasks .
Outcome:	This tutorial focuses on the benefits of pre-trained language models and how to use them in NLP tasks.

Copied to clipboard

Zeliang Zhang, Xiaodong Liu, Hao Cheng, Chenliang Xu, Jianfeng Gao

Challenge:	Large Language Models (LLMs) have outstanding performance by learning a large number of model parameters on large amounts of data.
Approach:	They propose a method of grouping and pruning similar experts to improve the model’s parameter efficiency by a range of natural language tasks.
Outcome:	The proposed method outperforms other model pruning methods on a range of natural language tasks.

PLATO-XL: Exploring the Large-scale Pre-training of Dialogue Generation (2022.findings-aacl)

Copied to clipboard

Siqi Bao, Huang He, Fan Wang, Hua Wu, Haifeng Wang, Wenquan Wu, Zhihua Wu, Zhen Guo, Hua Lu, Xinxian Huang, Xin Tian, Xinchao Xu, Yingzhan Lin, Zheng-Yu Niu

Dialogue Systems & Conversational Agents

Challenge:	Experimental results show PLATO-XL achieves state-of-the-art results across multiple conversational tasks.
Approach:	They propose to train PLATO-XL models with up to 11 billion parameters, trained on Chinese and English social media conversations.
Outcome:	The proposed model achieves state-of-the-art on multiple conversational tasks, verifying its potential as a foundation model of conversational AI.

Sparse Universal Transformer (2023.emnlp-main)

Copied to clipboard

Shawn Tan, Yikang Shen, Zhenfang Chen, Aaron Courville, Chuang Gan

Fine-Tuning ASR models for Very Low-Resource Languages: A Study on Mvskoke (2024.acl-srw)

Challenge:	Existing models that use VTs as their backbone model are based on UTs that share parameters across layers and have better compositional generalization.
Approach:	They propose to use Sparse Mixture of Experts to reduce UT's computation complexity while retaining its parameter efficiency and generalization ability.
Outcome:	The proposed model achieves strong generalization results on formal language tasks and impressive parameter and computation efficiency on standard natural language benchmarks.

Copied to clipboard

Julia Mainzinger, Gina-Anne Levow

Cascaded Head-colliding Attention (2021.acl-long)

Challenge:	Recent advances in multilingual models for automatic speech recognition (ASR) have been able to achieve a high accuracy for languages with extremely limited resources.
Approach:	They examine the parameter efficiency of training an adapter for the Mvskoke language, an indigenous language of America.
Outcome:	The proposed model is parameter efficient and gives higher accuracy for a relatively small amount of data.

Copied to clipboard

Lin Zheng, Zhiyong Wu, Lingpeng Kong

TeRA: Vector-based Random Tensor Network for High-Rank Adaptation of Large Language Models (2026.acl-long)

Challenge:	Existing frameworks for natural language processing ignore interactions among different heads, which wastes the capacity of the model.
Approach:	They propose a model which explicitly models interactions between attention heads through a hierarchical variational distribution.
Outcome:	The proposed model outperforms the baseline model on Wikitext-103 and WMT14 EN-DE on language modeling and translation tasks.

Copied to clipboard

Yuxuan Gu, Wuyang Zhou, Giorgos Iacovides, Danilo Mandic

Knowledge Graph Embedding with Atrous Convolution and Residual Learning (2020.coling-main)

Challenge:	Low-Rank Adaptation (LoRA) methods have significantly reduced the number of trainable parameters needed in fine-tuning large language models.
Approach:	They propose a vector-based random Tensor network for high-Rank Adaptation method that achieves high-rank weight updates while retaining parameter efficiency.
Outcome:	The proposed method outperforms existing PEFT methods while keeping low-rank parameters.

Copied to clipboard

Feiliang Ren, Juchen Li, Huihui Zhang, Shilei Liu, Bochao Li, Ruicheng Ming, Yujia Bai

A Lightweight Mixture-of-Experts Neural Machine Translation Model with Stage-wise Training Strategy (2024.findings-naacl)

Challenge:	Existing knowledge graph embedding methods are complex and require time for training and inference.
Approach:	They propose an atrous convolution based knowledge graph embedding method that increases feature interactions by using atrous . they evaluate method on six benchmark datasets with different evaluation metrics .
Outcome:	The proposed method achieves better results on six benchmark datasets than state-of-the-art methods on most evaluation metrics.

Copied to clipboard

Fan Zhang, Mei Tu, Song Liu, Jinyao Yan

Text Generation

Challenge:	Using mixture-of-experts (MoE) to deal with language heterogeneity is a challenge in neural machine translation (NMT).
Approach:	They propose a lightweight MoE-based NMT model that is trained via an elaborate stage-wise training strategy.
Outcome:	The proposed model achieves stable improvements in translation tasks by introducing fewer extra parameters compared to baseline models.

PRoLoRA: Partial Rotation Empowers More Parameter-Efficient LoRA (2024.acl-long)

Copied to clipboard

Sheng Wang, Boyang Xue, Jiacheng Ye, Jiyue Jiang, Liheng Chen, Lingpeng Kong, Chuan Wu

MoDULA: Mixture of Domain-Specific and Universal LoRA for Multi-Task Learning (2024.emnlp-main)

Challenge:	Partially Rotation-enhanced Low-Rank Adaptation (PRoLoRA) is an intra-layer sharing mechanism that circumvents the drawbacks of peer parameter-sharing methods.
Approach:	They propose a partially rotation-enhanced low-rank adaptation (PRoLoRA) that shares four components to reduce the cost of LoRA and improves model capacity.
Outcome:	Empirical results show that PRoLoRA outperforms LoRA on multiple instruction tuning datasets.

Copied to clipboard

Yufei Ma, Zihan Liang, Huangyu Dai, Ben Chen, Dehong Gao, Zhuoran Ran, Wang Zihan, Linbo Jin, Wen Jiang, Guannan Zhang, Xiaoyan Cai, Libin Yang

ModularMoE: Fast LLM Customization with Parameter-Sharing Mixture-of-Experts for Low-Resource Settings (2026.findings-acl)

Challenge:	Recent advances in open-source Large Language Models (LLMs) have achieved notable successes in natural language processing.
Approach:	They propose a Parameter Efficient Fine-Tuning paradigm for improved fine-tuning and parameter efficiency in multi-task learning.
Outcome:	The proposed model outperforms existing methods on multi-task learning while reducing training costs by over 80% without losing general capability.

Copied to clipboard

Jiaxing Liu, Qi Qi, Haifeng Sun, Dunjun Li, Zirui Zhuang, Bo He, Xiang Yang, Cong Liu, Jianxin Liao, Jingyu Wang

Random Entity Quantization for Parameter-Efficient Compositional Knowledge Graph Representation (2023.emnlp-main)

Challenge:	Large Language Models impose significant computational and storage burdens on personal devices . existing customization approaches incur excessive computational costs or lead to suboptimal performance .
Approach:	They propose a training framework that converts pre-trained LLMs into parameter-sharing MoE models for lightweight deployment.
Outcome:	The proposed training framework outperforms state-of-the-art training frameworks at the same sparsity level while delivering up to 2.71 inference speedup.

Copied to clipboard

Jiaang Li, Quan Wang, Yi Liu, Licheng Zhang, Zhendong Mao

FedReFT: Federated Representation Fine-Tuning with All-But-Me Aggregation (2026.findings-eacl)

Challenge:	Existing approaches to learning on Knowledge Graphs (KGs) are not critical for learning on KGs.
Approach:	They propose an alternative approach to represent entities by composing entity-corresponding codewords matched from predefined small-scale codebooks.
Outcome:	The proposed approach achieves similar results to existing methods.

Copied to clipboard

Fatema Siddika, Md Anwar Hossen, Juan Pablo Munoz, Tanya G. Roosta, Anuj Sharma, Ali Jannesari

Continual Sequence Generation with Adaptive Compositional Modules (2022.acl-long)

Challenge:	Representation Fine-Tuning (ReFT) adapts large pre-trained models by updating only a small subset of parameters.
Approach:	They propose a method that uses sparse intervention layers to steer hidden representations directly to capture rich semantic information.
Outcome:	The proposed approach outperforms PEFTs on commonsense reasoning, arithmetic reasoning, and GLUE benchmarks while maintaining a high parameter efficiency.

Copied to clipboard

Yanzhe Zhang, Xuezhi Wang, Diyi Yang

Text Generation

Challenge:	Existing work on continual sequence generation relies on reuse of existing parameters to learn new tasks or blindly adds new parameters for each new task.
Approach:	They propose to use continual sequence generation to reuse existing parameters for new tasks . current continuous learning models often forget knowledge of dissimilar tasks if data distributions shift .
Outcome:	The proposed framework outperforms state-of-the-art models on various sequences of generation tasks.

Multi-Path Transformer is Better: A Case Study on Neural Machine Translation (2022.findings-emnlp)

Copied to clipboard

Ye Lin, Shuhan Zhou, Yanyang Li, Anxiang Ma, Tong Xiao, Jingbo Zhu

MoDE: Effective Multi-task Parameter Efficient Fine-Tuning with a Mixture of Dyadic Experts (2025.findings-naacl)

Challenge:	Extensive experiments on 12 WMT tasks show that shallower multi-path models can achieve similar or even better performance than the deeper model.
Approach:	They propose to use a parameter-efficient multi-path structure to fuse features extracted from different paths to achieve better performance.
Outcome:	The proposed model can achieve better performance with the same number of parameters than the deeper model.

Copied to clipboard

Lin Ning, Harsh Lara, Meiqi Guo, Abhinav Rastogi

Dual Complex Number Knowledge Graph Embeddings (2024.lrec-main)

Challenge:	Recent efforts have explored mixtures of LoRA modules for multi-task settings, but this study reveals redundancy in the down-projection matrix of these architectures.
Approach:	They propose a method to share down-projection matrix across tasks and employ atomic rank-one adapters coupled with routers that allow more sophisticated task-level specialization.
Outcome:	The proposed method outperforms state-of-the-art models on a SNI benchmark and provides a practical solution for deploying lightweight models.

Copied to clipboard

Yao Dong, Qingchao Kong, Lei Wang, Yin Luo

From Experts to Bases: Orthogonal Subspace Mixture for Continual Multimodal Instruction Tuning (2026.acl-long)

Challenge:	Existing knowledge graph embedding methods fail to model non-commutative composition patterns . extending to such sophisticated spaces increases the amount of parameters, which greatly reduces the parameter efficiency.
Approach:	They propose a new knowledge graph embedding method that maps entities to the dual complex number space and represents relations as rotations in 2D space via dual complex multiplication.
Outcome:	Experiments on multiple benchmark knowledge graphs show that the proposed method improves link prediction and path query answering.

Copied to clipboard

Pei Chen, Xilai Wang, null Shiqixu, Zejian Li, Lingyun Sun

DenseLoRA: Dense Low-Rank Adaptation of Large Language Models (2025.acl-long)

Challenge:	Existing parameter-efficient approaches to multimodal Continual Instruction Tuning suffer from knowledge interference and inefficient capacity expansion, limiting scalability.
Approach:	They propose a framework for multimodal Continual instruction tuning that decomposes adaptation weights into a globally shared pool of orthonormal bases to capture task-invariant knowledge.
Outcome:	Experiments show that MoBLoRA outperforms state-of-the-art methods while maintaining superior parameter efficiency.

Copied to clipboard

Lin Mu, Xiaoyu Wang, Li Ni, Yang Li, Zhize Wu, Peiquan Jin, Yiwen Zhang

CoLo: A Contrastive Learning Based Re-ranking Framework for One-Stage Summarization (2022.coling-1)

Challenge:	Low-rank adaptation (LoRA) is an efficient approach for adapting large language models (LLMs) but many of the weights in these matrices are redundant, leading to inefficiencies in parameter utilization.
Approach:	They propose a low-rank adaptation approach that fine-tunes two low-ranked matrices and adapts them through a dense low-Rank matrix, improving parameter utilization and adaptation efficiency.
Outcome:	The proposed approach achieves 83.8% accuracy with only 0.01% of trainable parameters compared to LoRA's 80.8% with 0.70% of trainability parameters on LLaMA3-8B.

Copied to clipboard

Chenxin An, Ming Zhong, Zhiyong Wu, Qin Zhu, Xuanjing Huang, Xipeng Qiu

Stronger, Lighter, Better: Towards Life-Long Attribute Value Extraction for E-Commerce Products (2024.findings-acl)

Challenge:	Existing methods for extractive and abstractive summarization use token-level or sentence-level training objectives.
Approach:	They propose a Contrastive Learning based re-ranking framework for one-stage summarization called CoLo.
Outcome:	The proposed framework boosts extractive and abstractive results on CNN/DailyMail benchmarks while maintaining inference efficiency.

Copied to clipboard

Tao Zhang, Chenwei Zhang, Xian Li, Jingbo Shang, Hoang Nguyen, Philip Yu

Are ELECTRA’s Sentence Embeddings Beyond Repair? The Case of Semantic Textual Similarity (2024.findings-emnlp)

Challenge:	Existing models for attribute value extraction struggle for parameter efficiency and reliability due to data contamination and catastrophic forgetting.
Approach:	They propose to decouple product type and attribute to promote de-contamination and parameter efficiency while scaling up.
Outcome:	The proposed model achieves state-of-the-art performance with affordable parameter size, least historical knowledge forgetting, and greatest robustness against noises.

Copied to clipboard

Ivan Rep, David Dukić, Jan Šnajder

Capability Decomposition for Unified Information Extraction via Hierarchical Mixture-of-Experts (2026.acl-long)

Challenge:	ELECTRA's sentence embeddings are poorer than BERT's, resulting in a significant drop in performance for semantic textual similarity (STS).
Approach:	They propose to use a truncated model fine-tuning method to repair the embeddings by reducing the number of parameters and producing smaller embeddables.
Outcome:	The proposed method improves Spearman correlation coefficient by over 8 points while increasing parameter efficiency on the STS Benchmark.

Copied to clipboard

Jing Zhou, Peng Wang, Wenjun Ke, Jiajun Liu, Yao He

LaMDA: Large Model Fine-Tuning via Spectrally Decomposed Low-Dimensional Adaptation (2024.findings-emnlp)

Challenge:	Existing methods for IE tasks suffer from inconsistent schema representation and implicitly intermediate reasoning . UC-UIE adopts a low-rank adapted hierarchical Mixture-of-Experts adapter for UIE tasks .
Approach:	They propose a framework that decomposes IE reasoning into three universal capabilities . UC-UIE adopts a low-rank Adaptation adapter to fine-tune LLMs for IE tasks .
Outcome:	The proposed framework outperforms full-parameter tuning methods with 1.24% trainable parameters and outperformed existing methods in generalization and interpretability.

Copied to clipboard

Seyedarmin Azizi, Souvik Kundu, Massoud Pedram

Exploring Geometric Representational Disparities between Multilingual and Bilingual Translation Models (2024.lrec-main)

Challenge:	Low-rank adaptation (LoRA) fine-tunes large language models due to its significant reduction in trainable parameters, but its backward updates require storing high-dimensional intermediate activations and optimizer states, requiring high peak GPU memory.
Approach:	They propose a low-dimensional adaptation approach to fine-tune large language models which freezes a first projection matrix while introducing a lower-dimensional trainable square matrix.
Outcome:	The proposed approach reduces trainable parameters and peak GPU memory footprint while preserving low-dimensional trainable square matrix.

Copied to clipboard

Neha Verma, Kenton Murray, Kevin Duh

LayoutDIT: Layout-Aware End-to-End Document Image Translation with Multi-Step Conductive Decoder (2023.findings-emnlp)

Challenge:	Existing work shows that limited modeling capacity is a major contributor to reduced performance in multilingual models.
Approach:	They investigate the isotropy of multilingual model decoder representations using intrinsic dimensionality and IsoScore to measure how they utilize the dimensions in their underlying vector space.
Outcome:	The proposed model decoder representations are less isotropic and occupy fewer dimensions than bilingual models.

Copied to clipboard

Zhiyang Zhang, Yaping Zhang, Yupu Liang, Lu Xiang, Yang Zhao, Yu Zhou, Chengqing Zong

Multimodality

Challenge:	Existing methods struggle to capture the visual layout in complex document images.
Approach:	They propose to integrate layout knowledge into document image translation by using a layout-aware encoder and a multi-step conductive decoder to achieve the translation step by step.
Outcome:	The proposed model outperforms state-of-the-art methods with better parameter efficiency.

MoKA:Parameter Efficiency Fine-Tuning via Mixture of Kronecker Product Adaption (2025.coling-main)

Copied to clipboard

Beiming Yu, Zhenfei Yang, Xiushuang Yi

Learning Continuous Temporal Dynamics on Symplectic Manifolds for Temporal Knowledge Graph Embedding (2026.findings-acl)

Challenge:	Low-Rank Adaptation (LoRA) is one of the most popular PEFT methods . low-rank update mechanism of LoRA somewhat limits its ability to approximate full-parameter fine-tuning during training process.
Approach:	They propose a parameter-efficient fine-tuning framework that combines Kronecker product with the Mixture-of-Experts method to achieve parameter efficiency and better model performance.
Outcome:	The proposed framework outperforms existing methods on the GLUE benchmark and instruction tuning tasks for large language models.

Copied to clipboard

Jiang Li, Zehua Duo, Tian Lan, Feilong Bao, Guanglai Gao, Xiangdong Su

CoMoL: Efficient Mixture of LoRA Experts via Dynamic Core Space Merging (2026.findings-acl)

Challenge:	Existing methods for temporal knowledge graph embedding lack explicit structural constraints for continuous-time dynamics.
Approach:	They propose a Temporal Knowledge Graph Embedding framework that embeds temporal dynamics into a symplectic phase space.
Outcome:	The proposed framework achieves competitive performance with lower embedding dimensions.

Copied to clipboard

Jie Cao, Zhenxuan Fan, Zhuonan Wang, Tianwei Lin, Ziyuan Zhao, Rolan Yan, Wenqiao Zhang, Feifei Shao, Hongwei Wang, Jun Xiao, Siliang Tang

DueT: Image-Text Contrastive Transfer Learning with Dual-adapter Tuning (2023.emnlp-main)

Challenge:	Existing PEFT methods suffer from limited parameter efficiency and coarse-grained adaptation due to proliferation of LoRA experts and instance-level routing.
Approach:	They propose a new MoE-LoRA framework that incorporates expert diversity, parameter efficiency, and fine-grained adaptation.
Outcome:	The proposed framework outperforms existing methods on multiple tasks while maintaining parameter efficiency.

Copied to clipboard

Taku Hasegawa, Kyosuke Nishida, Koki Maeda, Kuniko Saito

TalkLoRA: Communication-Aware Mixture of Low-Rank Adaptation for Large Language Models (2026.acl-long)

Challenge:	Comparative learning models for vision and language models are gaining popularity . dueT trains only adapters inserted into pre-trained image and text encoders .
Approach:	They propose a transfer learning method for vision and language models built by contrastive learning that trains only adapters inserted into the frozen image and text encoders.
Outcome:	The proposed method outperforms fine-tuning, and the LoRA-based adapter method in English and Japanese domains.

Copied to clipboard

Lin Mu, Haiyang Wang, Li Ni, Lei Sang, Zhize Wu, Peiquan Jin, Yiwen Zhang

From Pseudo-Balancing to True Specialization: Memory-Aware Routing for Mixture-of-Experts (2026.findings-acl)

Challenge:	Existing LoRA methods assume that experts operate independently, leading to unstable routing, expert dominance.
Approach:	They propose a communication-aware MoELoRA framework that relaxes this assumption by introducing expert-level communication prior to routing.
Outcome:	The proposed framework outperforms vanilla LoRA and MoELoRA on diverse language understanding tasks while maintaining expert dominance.

Copied to clipboard

Peixuan Hou, Yunbo Hou, Bin Chen, LI He, Jian Xu, Weiping Li, Bo Zheng, Guojie Song

MoA: Heterogeneous Mixture of Adapters for Parameter-Efficient Fine-Tuning of Large Language Models (2026.acl-long)

Challenge:	Existing methods to optimize expert-centered load balancing fail to account for pseudo-balance phenomenon . severe knowledge overlap among experts leads to redundant representations and inefficient parameter utilization .
Approach:	They propose a method that prioritizes expert utilization over semantic alignment . they use memory-aware routing to ensure expert load balancing is consistent .
Outcome:	Experimental results show that MAR improves expert specialization by 35% and accuracy by 2%-25% . MAR matches baseline performance with only half the experts .

Copied to clipboard

Jie Cao, Tianwei Lin, Bo Yuan, Rolan Yan, Hongyang He, Wenqiao Zhang, Juncheng Li, Dongping Zhang, Siliang Tang, Yueting Zhuang

MoEC: A Memory-Routed Mixture-of-Experts Controller for Adaptive Minecraft Control (2026.acl-long)

Challenge:	Existing methods for parameter-efficient fine-tuning (PEFT) are limited by computational costs and performance degradation.
Approach:	They propose a method that integrates Low-Rank Adaptation and Mixture-of-Experts (MoE) they propose combining expert load imbalance and representation collapse to improve LLM performance .
Outcome:	The proposed method outperforms homogeneous MoE-LoRA architectures in performance and parameter efficiency.

Copied to clipboard

Hui Wu, Chao Xu, Jianghui Wang, Ziqiong Liu, Dong Li, Yiwei Dai, Emad Barsoum

Parameter-Efficient Transfer Learning for End-to-end Speech Translation (2024.lrec-main)

Challenge:	Existing systems rely on a monolithic policy to execute subgoals across varying contexts, causing inconsistent outcomes and scaling only partially mitigates.
Approach:	They propose a memory-routed mixtureof-experts controller for Adaptive Minecraft Control that routes via a subgoal-indexed expert memory and regulates capacity through failure-triggered expert growth and redundancy-aware consolidation.
Outcome:	The proposed controller shows significant gains in adaptability, robustness, and execution consistency over strong baselines.

Copied to clipboard

Yunlong Zhao, Kexin Wang, Qianqian Dong, Tom Ko

Speech & Audio in NLP

Challenge:	Existing approaches to improve end-to-end speech translation are limited by the availability of labeled data.
Approach:	They propose a method which utilizes two lightweight adaptation techniques to modulate Attention and the Feed-Forward Network while preserving the capabilities of pre-trained models.
Outcome:	The proposed method outperforms baseline models and significantly improves performance in low-resource settings.

Enhancing Chain-of-Thought Reasoning with Critical Representation Fine-tuning (2025.acl-long)

Copied to clipboard

Chenxi Huang, Shaotian Yan, Liang Xie, Binbin Lin, Sinan Fan, Yue Xin, Deng Cai, Chen Shen, Jieping Ye

Riemannian Optimization for LoRA on the Stiefel Manifold (2025.findings-emnlp)

Challenge:	Representation Fine-tuning (ReFT) is a proposed method for improving parameter efficiency . however, it yields suboptimal performance, as fixed-position representations have uncertain impact on outputs .
Approach:	They propose a method that fine-tunes critical representations in a low-rank linear subspace while freezing the base model.
Outcome:	The proposed method improves accuracy of LLaMA-2-7B and ReFT by 18.2 and 3.8 on GSM8K.

Copied to clipboard

JuneYoung Park, Minjae Kang, Seongbae Lee, Haegang Lee, Seongwan Kim, Jaeho Lee

Label-semantics Aware Generative Approach for Domain-Agnostic Multilabel Classification (2025.findings-acl)

Challenge:	powerful, large language models (LLMs) present significant fine-tuning challenges due to their size.
Approach:	They propose to optimize LoRA’s B matrix by imposing explicit orthogonality constraints that achieve near-perfect orthogonal and full effective rank.
Outcome:	The proposed method outperforms AdamW and LoRA in terms of parameter efficiency and representational capacity.

Copied to clipboard

Subhendu Khatuya, Shashwat Naidu, Saptarshi Ghosh, Pawan Goyal, Niloy Ganguly

Information Retrieval

Challenge:	Existing approaches to multi-label text classification are limited by textual data.
Approach:	They propose a domain-agnostic generative model framework for multi-label text classification that generates predefined label descriptions and matches them to predefined labels.
Outcome:	The proposed model achieves 13.94% and 24.85% performance over all datasets.

Localized Low-Rank Adaptation within Clustered Parameter Subspaces (2026.acl-long)

Copied to clipboard

Jiahao Xiong, Yihe Liu, Xianming Hu, Hongbo Zhao, Nuoyi Chen, Jie Zhang, Kai Zhang

BMRetriever: Tuning Large Language Models as Better Biomedical Text Retrievers (2024.emnlp-main)

Challenge:	Low-Rank Adaptation (LoRA) for large language models has been successful in various domains.
Approach:	They propose to perform low-rank updates within clustered parameter subspaces . they group rows/columns of update matrix into locally coherent, uncorrelated subspace blocks .
Outcome:	Empirical results show that low-rank Adaptation (LoRA) is better than global adaptations in various domains.

Copied to clipboard

Ran Xu, Wenqi Shi, Yue Yu, Yuchen Zhuang, Yanqiao Zhu, May Dongmei Wang, Joyce Ho, Chao Zhang, Carl Yang

Language Fusion for Parameter-Efficient Cross-lingual Transfer (2025.acl-long)

Challenge:	Developing effective biomedical retrieval models is important for excelling at knowledge-intensive biomedically tasks but still challenging due to the lack of sufficient publicly annotated biomedic data and computational resources.
Approach:	They propose a series of dense retrievers for enhancing biomedical retrieval via unsupervised pre-training on large biomedically corpora, followed by instruction fine-tuning on a combination of labeled datasets and synthetic pairs.
Outcome:	Experiments on 5 biomedical tasks across 11 datasets confirm the performance of the retrieval model on various biomedically demanding tasks.

Copied to clipboard

Philipp Borchert, Ivan Vulić, Marie-Francine Moens, Jochen De Weerdt

Question Answering

Challenge:	Limited availability of multilingual text corpora for pretraining results in poor performance on downstream tasks due to undertrained representation spaces for languages other than English.
Approach:	They propose a method that integrates source and target language representations within low-rank (LoRA) adapters using lightweight linear transformations to enhance representation quality and transfer performance for languages other than English.
Outcome:	The proposed method improves representation quality and performance for languages other than English while maintaining parameter efficiency.

From Outcome to Process: Optimizing MoE Load Balancing with MCTS (2026.findings-acl)

Copied to clipboard

Wenjun Ke, Hengyuan Xu, Ziyu Shang, Yao He, Jiahao Wang, Zijie Xu, Peng Wang, Yuhang Lou, Jiajun Liu

Grouped Adaptive Weight Sharing (GAWS): An Inference-Efficient Adaptation Method for Large Language Models (2026.findings-acl)

Challenge:	Existing balancing strategies focus on constraining the final distribution of expert usage, but overlook the routing decisions made at each layer.
Approach:	They propose a three-stage framework that leverages process-level rewards to guide balanced expert routing.
Outcome:	Extensive experiments show that LayerMoE improves the performance of state-of-the-art LoRA-MoA baselines, yielding an average accuracy gain of 1.39%.

Copied to clipboard

Eman Alsuradi, Junhyun Lee, Kyenghun Lee, Hyeonmok Ko, Fahed Jubair

Union-of-Experts: Neurons in Mixture-of-Experts are Secretly Routers (2026.acl-long)

Challenge:	Low-Rank Adaptation (LoRA) is a new approach to fine-tuning large language models . adapters are lightweight, task specific modules that can be used for adapters in latency-sensitive settings.
Approach:	They propose a low-rank adapter with a weight sharing mechanism that reduces latency by 40% . they analyze LoRA adapters on GPUs and identify segmented function calls as the primary source of latency.
Outcome:	The proposed adapter reduces latency to about 40% of the gap between the unmerged LoRA and the base model while maintaining parameter efficiency and comparable accuracy.

Copied to clipboard

Songhao Wu, Ang Lv, Ruobing Xie, Samm Sun, Di Wang, Rui Yan, Yankai Lin

BEFT: Bias-Efficient Fine-Tuning of Language Models in Low-Data Regimes (2026.acl-long)

Challenge:	Mixture-of-Experts (MoE) models rely on an external router to assign tokens to experts, resulting in suboptimal performance.
Approach:	They propose an MoE variant that performs "expert-autonomous routing" by pre-designating a fraction of neurons within each expert as "routing neurons" they pre-train UoE models with up to 3B parameters and show they outperform traditional MoEs with matched efficiency.
Outcome:	The proposed model outperforms existing models with 3B parameters and provides valuable insights into expert-autonomous selection and the broader routing mechanisms of MoE models.

Copied to clipboard

Baichuan Huang, Ananth Balashankar, Amir Aminifar